Sensor fusion of motion data obtained from portable electronic devices

ABSTRACT

A method for determining an orientation of a portable or mobile electronic device includes determining an orientation of the device using at least a first inertial motion sensor (e.g., a gyroscope) with which the portable electronic device is equipped. A correction factor is provided to the orientation of the electronic device using a feedback control signal based on motion data obtained from at least a second inertial motion sensor (e.g. an accelerometer) to reduce drift in motion data obtained from the first inertial sensor. Responsive to a loss of valid motion data from the first inertial motion sensor, a rate at which the correction factor is provided to the orientation of the portable electronic device is increased.

BACKGROUND

Portable or mobile electronic devices are used in a wide variety ofapplications and environments. The ubiquity of such devices as mobilephones, digital cameras, personal digital assistants (PDAs) handheldmusic and media players, portable video game devices and controllers,mobile interne devices (MIDs), personal navigation devices (PNDs), andother portable devices attest to the popularity and desire for thesetypes of devices. Motion sensors, such as inertial sensors likeaccelerometers, gyroscopes, can be used in handheld electronic devices.For instance, accelerometers can be used for measuring accelerationincluding gravity, and gyroscopes can be used for measuring angularvelocity of a portable electronic device. This information can be usedto determine the spatial orientation and position of the device. Inaddition to inertial sensors, portable electronic devices also may beequipped with other types of sensors such as a magnetometer, forinstance, which can be used as a compass, and GPS, which can be used todetermine the device's position on Earth.

Many applications could benefit from the accurate determination of thespatial orientation and position of portable electronic devices. Forinstance, some portable device may use the orientation or relativemotion of the device as a source of user input. Video game consolecontrollers, for example, may use accelerometers to detect motion of thehand controller that is used to provide input to a game. Other gamecontrollers may be used as a three dimensional gesture input device(e.g., a virtual golf club). Other applications that may use suchspatial information include augmented reality applications, which canprovide real-time visualization with respect to the physical positionand orientation of the portable device. In yet another application,image stabilization is often a significant feature in even low- ormid-end digital cameras, where lens or image sensors are shifted tocompensate for hand jittering measured by a gyroscope.

SUMMARY

In one implementation, a gyroscope is used as the primary source of datato obtain the orientation of a portable electronic device. Advantages ofthis approach include the rapid response and low noise that is found intypical gyroscopes. However, because of drift in the data and theoccasional loss of valid gyroscope data, the instantaneous orientationdata obtained from the gyroscope is supplemented by a periodiccorrection factor using data obtained from an accelerometer and possiblya magnetometer (if available). The periodic correction may be appliedusing a feedback control mechanism such as aproportional-integral-derivative (PID) feedback control mechanism. Inone particular implementation, the proportional and integral responsesof the PID control mechanism are employed, but not the derivativeresponse. In another particular implementation, only the proportionalresponse of the PID control mechanism is employed.

In one implementation, the PID control mechanism may enter a fastcorrection mode when valid gyroscope data is unavailable. In this modethe proportional response can be increased and the integral responsesuppressed for a short period of time so that it responds more quickly.In this way the orientation can be primarily tracked using theaccelerometer and magnetometer. Once the gyroscope data again becomesavailable, the PID control mechanism can exit the fast correction mode.

In other implementations the amount of low pass filtering that theaccelerometer data undergoes to remove noise may be varied because a lagis introduced to the accelerometer data which increases with increasedfiltering. Since lag becomes more problematic as the electronic deviceundergoes more rapid rotational motion, the amount of filteringperformed on the accelerometer data is decreased as the rotational speedas measured by the gyroscope increases.

This Summary is provided to introduce a selection of concepts in asimplified form that are further described below in the DetailedDescription. This Summary is not intended to identify key features oressential features of the claimed subject matter, nor is it intended tobe used as an aid in determining the scope of the claimed subjectmatter.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a perspective view of one example of a portable electronicdevice.

FIG. 2 is a block diagram illustrating one example of a portableelectronic device such as the device shown in FIG. 1.

FIG. 3 shows a frame of reference that may be used to express theorientation of the portable electronic device in the Earth coordinatesystem.

FIG. 4 is a flowchart showing one example of a method for determiningthe orientation of a portable electronic device using data obtained fromtwo or more sensors located on the device.

DETAILED DESCRIPTION

FIG. 1 is a perspective view of one example of a portable electronicdevice 10. The device 10 can be held in one or more hands of a user tobe operated, and can include a variety of different functions, asdescribed below. In this example the device 10 includes a display screen16 a and physical buttons 6 as well as buttons 8 and 9 on one or bothsides of the device 10. Other embodiments of devices can be used, andcan include different and/or additional input and output devices, asdescribed below with respect to FIG. 2.

The device 10 can be moved by the user in space, and this movement isdetected by motion sensors of the device as detailed below. As referredto herein, rotation of the device 10 can include pitch, roll, and yawabout the various rotational axes, as shown in FIG. 1. These axes can bedefined differently in other embodiments. Furthermore, linear motionscan be made along the linear axes x, y and z. Furthermore, these axescan be defined at various different positions on the device (for exampletranslated or rotated with respect to the axes shown in FIG. 1, orotherwise transposed into any other coordinate system (whetherrectangular, polar, or otherwise)), as appropriate for the hardware andsoftware used by the device 10.

FIG. 2 is a block diagram illustrating one example of a portableelectronic device such as the device 10 shown in FIG. 1. In someexamples the device is a mobile communications device such as a wirelesstelephone that also contains other functions, such as FDA and/or musicplayer functions. To that end the device may support any of a variety ofapplications, such as a telephone application, a video conferencingapplication, an e-mail application, an instant messaging application, ablogging application, a digital camera application, a digital videocamera application, a web browsing application, a digital music playerapplication, and/or a digital video player application. It may alsosupport applications that make use of data concerning the spatialorientation of the device, including but not limited to frames,augmented reality applications and the like. While the example in FIG. 1is depicted as a mobile communications device, the computing device moregenerally may by any of a wide variety of different devices such as alaptop computer, a tablet computer, a smart phone and a netbook, forexample.

The device 100 includes a memory unit 102 (which may include one or morecomputer readable storage media), a memory controller 122, one or moreprocessors (CPU's) 120, a peripherals interface 118, RF circuitry 108,audio circuitry 110, a speaker 111, a microphone 113, display system103, an input/output (I/O) subsystem 106, other input or control devices116, and an external port 124. These components may communicate over oneor more communication buses or signal lines. Along with the input orcontrol devices 116, the speaker 111, microphone 113 and display system103 form a user interface through which a user can enter and receivevarious types of information and can communicate with other individualsover communication networks using RF circuitry 108.

Memory unit 102 may include high-speed random access memory andnon-volatile memory, such as one or more magnetic disk storage devices,flash memory devices, or other non-volatile solid-state memory devices.Access to memory unit 102 by other components of the device 100, such asthe processor 120 and the peripherals interface 118, may be controlledby the memory controller 122. The peripherals interface 118 couples theinput and output peripherals of the device to the processor 120 andmemory unit 102. The one or more processors 120 run or execute varioussoftware programs and/or sets of instructions stored in memory unit 102to perform various functions for the device 100 and to process data. Insome examples the peripherals interface 118, the processor 120, and thememory controller 122 may be implemented on a single chip, such as achip 104. In other examples they may be implemented on separate chips.

The RF (radio frequency) circuitry 108 includes a receiver andtransmitter (e.g., a transceiver) for respectively receiving and sendingRF signals, also called electromagnetic signals. The RF circuitry 108converts electrical signals to/from electromagnetic signals andcommunicates with communications networks and other communicationsdevices via the electromagnetic signals. The RF circuitry 108 mayinclude well-known circuitry for performing these functions, includingbut not limited to an antenna system, an RF transceiver, one or moreamplifiers, a tuner, one or more oscillators, a digital signalprocessor, a CODEC chipset, a subscriber identity module (SIM) card,memory, and so forth. The RF circuitry 108 may communicate withnetworks, such as the Internet, also referred to as the World Wide Web(WWW), an intranet and/or a wireless network, such as a cellulartelephone network, a wireless local area network (LAN) and/or ametropolitan area network (MAN), and other devices by wirelesscommunication. The wireless communication may use any of a plurality ofcommunications standards, protocols and technologies, including but notlimited to Global System for Mobile Communications (GSM), Enhanced DataGSM Environment (EDGE), high-speed downlink packet access (HSDPA),wideband code division multiple access (W-CDMA), code division multipleaccess (CDMA), time division multiple access (TDMA), Bluetooth, WirelessFidelity (Wi-Fi) (e.g., IEEE 802.11a, IEEE 802.11b, IEEE 802.11a and/orIEEE 802.11n), voice over Internet Protocol (VoIP), Wi-MAX, a protocolfor email, instant messaging, and/or Short Message Service (SMS)), orany other suitable communication protocol, including communicationprotocols not yet developed as of the filing date of this document.

The audio circuitry 110, the speaker 111, and the microphone 113 form apart of the user interface provide an audio interface between a user andthe device 100. The audio circuitry 110 receives audio data from theperipherals interface 118, converts the audio data to an electricalsignal, and transmits the electrical signal to the speaker 111. Thespeaker 111 converts the electrical signal to human-audible sound waves.The audio circuitry 110 also receives electrical signals converted bythe microphone 113 from audible signals (i.e., sound waves). The speaker111 and microphone 113 are two examples of audio transducers that may beemployed in the mobile communications device. The audio circuitry 110converts the electrical signal to audio data and transmits the audiodata to the peripherals interface 118 for processing. Audio data may beretrieved from and/or transmitted to memory unit 102 and/or the RFcircuitry 108 by the peripherals interface 118. In some embodiments, theaudio circuitry 110 also includes a headset jack (not shown). Theheadset jack provides an interface between the audio circuitry 110 andremovable audio input/output peripherals, such as output-only headphonesor a headset with both output (e.g., a headphone for one or both ears)and input (e.g., a microphone).

The I/O subsystem 106 couples input/output peripherals on the device100, such as the display screen 112, sensor controller 149 and otherinput/control devices 116, to the peripherals interface 118. The I/Osubsystem 106 may include a display controller 156 and one or more inputcontrollers 160 for other input or control devices. The one or moreinput controllers 160 receive/send electrical signals from/to otherinput or control devices 116. The other input/control devices 116 mayinclude physical buttons (e.g. push buttons, rocker buttons, etc.),dials, slider switches, joysticks, click wheels, and so forth. In someexamples input controller(s) 160 may be coupled to any (or none) of thefollowing: a keyboard, infrared port, USB port, and a pointer devicesuch as a mouse.

The display screen 112 provides an input interface and an outputinterface between the device and a user. The display controller 156receives and/or sends electrical signals front/to the display screen112. The display screen 112 displays visual output to the user. Thevisual output may include graphics, text, icons, video, and anycombination thereof (collectively termed “graphics”).

The display screen 112 will generally include a suitable display such asan OLED display, PLED display, active matrix liquid crystal display,passive matrix liquid crystal display, electrophoretic display,cholesteric liquid crystal display, polymer dispersed liquid crystal andnematic liquid crystal display. In some implementations the displayscreen 112 may be a touch-screen display.

The device 100 also includes one or more environmental sensors, whichare in communication with peripherals interface 118 via sensorcontrollers 149. Such sensors may be used to capture the value ofvarious environmental parameters and in this example, include twoinertial sensors (a gyroscope 148 and accelerometer 142) and an optionalmagnetometer 144.

The device 100 also includes a power system 162 for powering the variouscomponents. The power system 162 may include a portable power supply(e.g., battery) and components used to receive power from an alternatingcurrent (AC) source, a power management system, a recharging system, apower failure detection circuit, a power converter or inverter and anyother components associated with the generation, management anddistribution of power in portable devices.

In some embodiments, the software components stored in memory unit 102may include an operating system 126, a communication module (or set ofinstructions) 128, a contact/motion module (or set of instructions) 130,a graphics module (or set of instructions) 132, a text input module (orset of instructions) 134, a Global Positioning System (GPS) module (orset of instructions) 135, a sound module 133 (or set of instructions)and applications (or set of instructions) 136.

The operating system 126 (e.g., Darwin, RTXC, LINUX, UNIX, OS X,Microsoft WINDOWS®, Android or an embedded operating system such asVxWorks) includes various software components and/or drivers forcontrolling and managing general system tasks (e.g., memory management,storage device control, power management, etc.) and facilitatescommunication between various hardware and software components. Thecommunication module (or set of instructions) 128 facilitatescommunication with other devices over one or more external ports 124 andalso includes various software components for handling data received, bythe RF circuitry 108 and/or the external port 124 (e.g., UniversalSerial Bus (USB), FIREWIRE, etc.).

The graphics module 132 includes various known software components forrendering and displaying graphics on the display screen 112, includingcomponents for changing the intensity of graphics that are displayed. Asused herein, the term “graphics” includes an object that can bedisplayed to a user, including without limitation text, web pages, icons(such as user-interface objects including soft keys), digital images,videos, animations and the like. The text input module (or set ofinstructions) 134, which may be a component of graphics module 132,provides soft keyboards for entering text in various applications (e.g.,contacts 137, e-mail 140, IM 141, blogging 142, browser 147, and anyother application that needs text input).

The GPS module 135 determines the location of the device and providesthis information for use in various applications (e.g., applicationsthat provide location-based services such as weather widgets, localyellow page widgets, and map/navigation widgets). It should be notedthat if such applications are employed and if as a result userinformation and location information is collected, such information willonly be collected and stored after notice has been provided to the userthat the collection of personal information may occur, for example, whensigning up to use a location-based service. The notice will alsoindicate that this information will not be shared with third parties,other than as may be needed to maintain or enhance the quality of theservice that is being provided. Other policies that are intended toprotect the user's privacy and enhance the quality of the userexperience may also be employed. Once the user is informed as to theterms of service, the user will be given an opportunity to consent tothe terms of service.

The message indicator control module 136 implements the functionalitydescribed herein to adjust a characteristic of one or more messageindicators in response to the ambient audio level in the device's localenvironment. The ambient audio level may be determined by the measuringthe audio level with the microphone 113 or another sensor dedicated tothis purpose. Additional functionality and implementations of themessage indicator control module 136 will be discussed below.

The message indicators employed by the communication device 100 mayinclude the speaker 111, display screen 112 and/or other messageindicators 140 such a dedicated ringer and a haptic feedback mechanism.If a speaker or ringer is employed as message indicator, itscharacteristic that may be adjusted includes its volume level, forexample. If a display is employed as a message indicator, itcharacteristic that may be adjusted includes its brightness, colorscheme, and/or graphics scheme. A haptic feedback mechanism is arrangedto provide tactile feedback to a user of the communication device, inthis case, for example, the haptic interface may be employed to vibratemobile communication device 100 in different was when different types ofmessages are received.

The applications 138 may include any combination of the followingillustrative modules: a contacts module, a telephone module; a videoconferencing module; an e-mail client module an instant messaging (IM)module; a blogging module; a camera module; an image management module;a video player module; a music player module; a browser module; a wordprocessing module; a voice recognition module; a calendar module; widgetmodules, which may include a weather widget, stocks widget, calculatorwidget, alarm clock widget, dictionary widget, and other widgetsobtained by the user, as well as user-created widgets. As describedabove, one or more of the applications may also employ spatialorientation data.

Each of the above identified modules and applications correspond to aset of instructions for performing one or more functions describedabove. These modules (i.e., sets of instructions) need not beimplemented as separate software programs, procedures or modules, andthus various subsets of these modules may be combined or otherwisere-arranged in various embodiments. In some embodiments, memory unit 102may store a subset of the modules and data structures identified above.Furthermore, memory unit 102 may store additional modules and datastructures not described above.

It should be appreciated that the device 100 is only one example of amobile communications device 100 and that the device 100 may have moreor fewer components than shown, may combine two or more components, or amay have a different configuration or arrangement of components. Thevarious components shown in FIG. 1 may be implemented in hardware,software or a combination of hardware and software, including one ormore signal processing and/or application specific integrated circuits.

As mentioned above, in one implementation the inertial sensors includeone or more rotational motion sensors such as gyroscope 148 and one ormore linear motion sensors such as accelerometers 142. Gyroscopes 148can sense or measure the angular velocity of the device 10 housing thegyroscopes 148. That is, gyroscopes output angular velocity in devicecoordinates. Gyroscopes generally have very good precision (e.g., 16bit) and quality (e.g., very low noise, very low latency), Gyroscopeshaving from one to three axes may be used, depending on the motion thatis desired to be sensed in a particular embodiment, with a three-axisgyroscope being typical. Some gyroscopes may be dynamically activated ordeactivated, for example to control power usage or adapt to motionprocessing needs. Some gyroscopes may have variable sensitivity or gain.The more sensitive the gyroscope, the narrower its useful operatingrange and vice-versa.

The angular velocity of the portable electronic device obtained from thegyroscope can be integrated to obtain the relative angular position ofthe device at any given time. This integration eventually produces adrift in the data due to approximations arising from discrete sampling(the faster the better), offset error (any constant offset can becalibrated out), noise (typically very low), quantization, rounding ofcalculations, gain error (which is generally minimized by a factorycalibration), saturation (reaching the maximum readable rotation rate ofthe gyroscope), etc. Given the overall quality of the gyroscope signal,this drift may be relatively small, but accumulates over time up to afew degrees per minutes), except in the case of saturation, when no datais available.

Because of its rapid response time and low noise, the gyroscope may beused as the primary or sole source of relative orientation data.However, because of its drift, among other problems, the gyroscope maybe supplemented with additional sources of motion data obtained from theother sensors.

The accelerometer 142 can sense or measure the linear acceleration ofthe device 10 (or portion thereof) housing the accelerometer 142. Anaccelerometer operates by measuring the deflection of a small masssuspended by springs. The natural frequencies of the dynamics of theaccelerometer are generally high and thus it responds quickly. Theaccelerometer outputs the sum of the linear acceleration in devicecoordinates and tilt due to gravity. That is, an accelerometer does notdistinguish between gravity and any additional acceleration of thedevice above and beyond gravity. From one to three axis accelerometerscan typically be provided, depending on the motion that is desired to besensed in a particular embodiment. Some accelerometers may bedynamically activated or deactivated, for example to control power usageor adapt to motion processing needs.

The magnetometer 144 measures the magnetic intensity experienced by thedevice 10 housing the magnetometer. When influenced by the Earth'smagnetic field this measurement indicates an absolute geographicdirection, in particular, since magnetometers indicate the direction ofmagnetic North (assuming the device does not experience any othermagnetic forces), they can provide a direction to the geographic northby knowing the magnetic declination, which varies with geographicallocation, is a known value which can be obtained from tables and thelike.

As noted above, the inertial sensors provide motion sensor data indevice coordinates. In order to obtain the orientation of the device inthe Earth's frame of reference, the motion sensor data is transformedinto the Earth coordinate system. One convention that may be used toexpress the Earth coordinate system is shown in FIG. 3.

Any of a variety of techniques may be used to transform the device framecoordinates to Earth frame coordinates, thereby obtaining the device'sorientation in Earth coordinates. For instance, Euler angles, rotationmatrices or quaternions may be used. These techniques are well and thusdo not need to be discussed in further detail herein.

Each of the aforementioned sensors has their own specific limitations.For instance, as noted above, while the gyroscope responds very rapidlyand its signal is low in noise, it does not provide absolute orientationinformation. Moreover, the output from the gyroscope drifts over timefor the reasons mentioned above. Accordingly, if the gyroscope were tobe used as the primary or sole source for orientation information, itwould need to be provided with an initial absolute orientation and itwould need periodic corrections to account for the drift. Also, thegyroscope data may need to undergo filtering with a high pass filter toreduce noise arising from its zero level bias.

One limitation of the accelerometer is that its signal to noise ratio isrelatively high and generally will need to undergo low-pass filtering.The same is generally true of the magnetometer as well. Unfortunately,the significant filtering that the accelerometer and magnetometerundergo introduces a time lag and the resulting data obtained from thesesensors will trail the gyroscope data when the device rotation is fast.

Thus, because of the aforementioned limitations none of the individualsensors is capable of extracting device orientation data with highprecision. One way to overcome this problem is to combine the data fromthe sensors in a way that uses the advantages of each. The process ofcombining data from different sensors is often referred to as sensorfusion.

One way to combine the sensor data which will be described below is touse the heavily filtered data from the accelerometer and possibly themagnetometer to obtain the initial absolute orientation of the deviceand then incrementally update this data using the gyroscope data. Thisapproach is advantageous because of the relatively fast response of thegyroscope. However, because of the drift in the gyroscope data, datafrom the accelerometer and possibly the magnetometer can be periodicallyused to correct for the drift in the gyroscope data. Because theaccelerometer and magnetometer data is only used for periodiccorrections, the lag in this data is of less concern. The rate at whichthe periodic corrections are applied to the gyroscope data may or maynot be the same as the gyroscope data rate. That is, if the gyroscopedata is obtained at a series of successive times, the periodiccorrections may be applied at each of those times or a subset of thosesuccessive periods of time.

Corrections to the gyroscope data may be performed using various sensorfusion techniques such as a Kalman filter or feedback controlmechanisms. For instance, a proportional-integral-derivative (PID)feedback control mechanism may be employed. In this approach the angularorientation estimation performed by integrating the angular velocitydata from the gyroscope is corrected by using a proportional andintegral feedback of the accelerator data and optionally themagnetometer data. The proportional response or function depends only onthe current error, which is the difference between the deviceorientation determined from the angular velocity data from the gyroscopeand the device orientation determined from the accelerometer andmagnetometer. The integral response or function, on the other hand,depends on the accumulated errors that have developed over time. In somecases the derivative response may not be used because this response canbe very sensitive to noise or a slow system response.

In the present context the integral response aims at correcting constanterrors such as the gyroscope bias. The proportional response or functionwill generally provide the primary correction, with the integralresponse providing a smaller correction. If the gyroscope bias can beeliminated in another way, then in some cases the integral response orfunction may even be eliminated, since it can cause the correctedgyroscope data to overshoot its desired value.

One problem that arises when the gyroscope is used as the primary sourceof instantaneous orientation data is that the data can be temporarilylost and thus a way is needed to recover from such losses. The loss of acontinuous stream of gyroscope data may occur for any of a variety ofdifferent reasons. For instance, data may be lost when the gyroscopesaturates (i.e., when the device rotates faster than its maximummeasurable rate along one of its axes), when the real-time behavior ofthe system is overloaded for too long a time and when the gyroscope istemporarily turned off to save power when no output is needed or whenpower is being used for other device functions such as establishing atelephone call or sending a message.

When gyroscope data is lost, or after a certain time has lapsed withoutreceipt of gyroscope data, the proportional response can be increasedfor a short period of time so that it responds more quickly. In this waythe orientation can be primarily tracked using the accelerometer andmagnetometer. In addition, the integral response may be completelysuppressed since long term corrections are not of concern during theperiod when gyroscope data is unavailable. This mode of operation may bereferred to as the fast correction mode of the feedback control system.After the gyroscope data once again becomes available, the controlsystem migrates back to its nominal values. The feedback control systemmay enter and exit the fast correction mode quite rapidly, with a rampup and decay time greater than linearity. For instance, the ramp up anddecay rates may be hyperbolic. In some cases, however, the migrationback to the nominal integral response may occur more gradually than thereturn to the nominal proportional response.

The use of accelerometer data as an input providing periodic errorcorrection to the gyroscope data has a number of limitations that can beaddressed. For example, as mentioned above, the accelerometer measuresthe sum of the linear acceleration in device coordinates and gravity.The resulting device acceleration vector will thus have a downwardcomponent due to gravity which is not easily separated from thenon-gravitational component of the acceleration vector. That is, adevice accelerating in a purely horizontal direction will have anacceleration vector tilted downward because of gravity. Accordingly, theaccelerometer is primarily useful to determine the orientation of thedevice when the device is not accelerating and can be used to determinethe orientation of the device by measuring the gravity vector alone.

Assuming the accelerometer is properly calibrated, it can provide anoutput value very close 1 g (the acceleration due to gravity) when thedevice is not undergoing any linear acceleration. If its output value issubstantially different from 1 g, the device is clearly undergoinglinear acceleration in addition to gravity. Accordingly, in someimplementations, the data from the accelerometer is only used asfeedback to correct the gyroscope data when its output value is close to1 g (e.g., within +/− of some tolerance value after the accelerationdata is filtered). Otherwise, if the accelerometer indicates a valuesubstantially different from gravity, its data is not used as acorrection factor to the gyroscope data until it returns to a value near1 g.

As mentioned above, another problem with the data from the accelerometeris that it needs to undergo low-pass filtering to remove noise, and as aresult, the accelerometer data may trail the gyroscope data. The amountby which the accelerometer data lags behind the gyroscope data dependson the amount of filtering that is performed; the more the data isfiltered the greater the lag (and visa versa). This problem can beaddressed by varying the amount of filtering that is performed on theaccelerometer data based on the data from the gyroscope. Specifically,the faster the device rotation as measured by the gyroscope, the morethe lag should be reduced. Thus, as the device rotation increases it maybe desirable to reduce the amount of filtering performed on theaccelerometer data in order to reduce the lag. Likewise, as the devicerotation decreases the amount of filtering performed on theaccelerometer data can be increased since more lag can be tolerated. Thevariation in the degree of filtering may be adjusted in a continuousmanner based on the rate of rotation of the device or it may be adjustedin one or more discrete increments. For example, one or more thresholdsmay be established such that if the rate of rotation increases above acertain specified threshold the amount of filtering decreases and if therate of rotation decreases below that threshold the amount of filteringincreases.

FIG. 4 is flowchart showing one example of a method for determining theorientation of a portable electronic device using data obtained from twoor more sensors located on the device. In this example three suchsensors are used: a gyroscope, an accelerometer, and a magnetometer.

The method begins at step 405 when a processor (e.g., processor 120 inthe device of FIG. 2) receives relative rotational data from thegyroscope. This data is sufficient to determine the relative orientationof the device. The gyroscope data is filtered with a high pass filter atstep 410 to remove any bias values that may arise from such things ascalibration errors, age, temperature and the like. The filteredgyroscope data is integrated at step 415 to obtain the relativeorientation of the device.

For the reasons previously discussed, the gyroscope data needs to beinitialized to get an absolute orientation of the device. The gyroscopedata also needs to be periodically corrected to account for drift andother error. Accordingly, a feedback process is applied at step 435 tothe device orientation data obtained in step 415. In this particularexample the feedback process is a PID feedback mechanism in which onlythe proportional and integral responses are employed. The feedback isbased on the absolute orientation information obtained from the sensorsother than the gyroscope. In this example those sensors are theaccelerometer and magnetometer and the data from each is obtained, atsteps 440 and 445, respectively.

However, if after filtering the gyroscope data it is determined at steps420 or 425 that the gyroscope data is invalid or unavailable,respectively, the processor enters a fast correction mode at step 430.More specifically, step 420 determines if the gyroscope is saturated orwithin a threshold amount of being saturated, in which case the deviceis rotating at too great a rate to be accurately measured by thegyroscope. In addition, step 425 determines if the gyroscope data hasbeen unavailable for too long a period of time. This includes delayswhich are caused by the gyroscope itself or what appears to be a delaydue to the initialization of the orientation determination method. Ifthe processor enters the fast correction mode at step 430, theproportional response in the feedback loop is rapidly increased and theintegral response is suppressed or deactivated. When the gyroscope datais once again available and deemed valid (e.g., no saturation), theproportional and integral responses migrate back to their nominalvalues.

Before the raw data obtained from the accelerometer and magnetometer canbe used in the feedback process, they may undergo various processing.For instance, in the example of FIG. 4, the accelerometer data obtainedat step 455 undergoes a variable low pass filtering process to reducenoise. That is, as discussed above, to reduce the lag between theaccelerometer data and the gyroscope data at fast rotational speeds, asthe device rotation rate increases as indicated by the gyroscope data,the amount of filtering performed on the accelerometer data is reducedin order to reduce the lag. Likewise, the amount of filtering isincreased as the device rotation rate determined by the gyroscopedecreases. In FIG. 4, the rate of device rotation determined by thegyroscope is determined at step 450 and provided as input data to thevariable pass filtering process at step 455.

Orientation data obtained from the magnetometer at step 445 alsoundergoes processing before it is used in the feedback process. Sincethe Earth's magnetic field may be represented as a dipole between theNorth and South Poles, the magnetic field measured by the magnetometerwill have an upward or downward components, depending on the hemispherein which the magnetometer is located. However, only the horizontalcomponent is pertinent in obtaining the direction of magnetic North.Accordingly, the horizontal projection of the magnetometer data isobtained in step 460 to determine magnetic North. Once magnetic Northhas been determined then true North can be obtained from this data atstep 465 by compensating for the declination, which can be obtained orestimated in any appropriate manner using, for instance, positioningtechnologies such as a GPS sensor if the device is so equipped, celltower triangulation or Wi-Fi location systems. If no other source ofposition data is available, the last known location of the device may beused in some cases.

As shown in FIG. 4, the processed data from the accelerometer and themagnetometer is input to the PID correction process of step 435. In thisway the accelerometer and magnetometer data are used to provide aperiodic correction to the orientation data obtained from the gyroscope.Next, in step 470, the corrected gyroscope orientation data is used todevelop a direction cosine matrix (DCM), which in this example is usedas the rotation matrix to express the orientation of the device in Earthcoordinates.

The DCM is a 3×3 matrix, where the columns represent unit vectors in thebody axes, projected along the reference axes. The element in the ithrow and jth column represents the cosine of the angle between the i-axisof the reference frame and j-axis of the body frame. Since the axes areorthogonal to each other, and their norm is to be equal to one, the DCMis an orthonormal matrix. Because of numerical errors, the DCM maygradually violate these orthogonal and normalization constraints.Accordingly, in step 470 the DCM undergoes a normalization process toensure that it remains a proper rotation matrix. Next, in step 475, theorientation or attitude of the device is directly obtained from the DCM.While in this example the orientation is expressed in terms of a DCM, aspreviously mentioned the orientation may be expressed in other terms aswell, using, for example, an Euler angle representation, a quaternionrepresentation, or other representation.

Finally, once the orientation of the device is known, its linearacceleration can be obtained at step 480 by subtracting the accelerationvector due to gravity from the acceleration vector obtained from theaccelerometer and compensated by the gyroscope. In addition, thecompensated rotational speed of the device can be obtained at step 485.

in some implementations sensor information may be available only fromthe accelerometer and magnetometer, but not the gyroscope. In this caseorientation data may still be obtained, albeit in a degraded mode. Inparticular, the gravitational acceleration component may not be able tobe separated from the lateral acceleration component in the dataobtained from the accelerometer. However, instead of being used simplyas a source of long term trend data, the accelerometer and magnetometerdata can be used more directly (typically with a small low-pass filterto keep the lag time short) to provide the attitude of the device. Thedirection of the accelerometer data may simply be assumed to be a resultof gravity and therefore is pointed to the ground. Thus, the lateralacceleration may only be based on the norm of the accelerometerreadings, not its orientation.

As used in this application, the terms “component,” “module,” “engine,”“system,” “apparatus,” “interface,” or the like are generally intendedto refer to a computer-related entity, either hardware, a combination ofhardware and software, software, or software in execution. For example,a component may be, but is not limited to being, a process running on aprocessor, a processor implementing processing logic, an object, anexecutable, a thread of execution, a program, and/or a computer. By wayof illustration, both an application running on a controller and thecontroller can be a component. One or more components may reside withina process and/or thread of execution and a component may be localized onone computer and/or distributed, between two or more computers.

Furthermore, the claimed subject matter may be implemented as a method,apparatus, or article of manufacture using standard programming and/orengineering techniques to produce software, firmware, hardware, or anycombination thereof to control a computer to implement the disclosedsubject matter. For instance, the claimed subject matter may beimplemented as a computer-readable storage medium embedded with acomputer executable program, which encompasses a computer programaccessible from any computer-readable storage device or storage media.For example, computer readable storage media can include but are notlimited to magnetic storage devices hard disk, floppy disk, magneticstrips . . . ), optical disks (e.g., compact disk (CD), digitalversatile disk (DVD) . . . ), smart cards, and flash memory devices(e.g., card, stick, key drive . . . ). Of course, those skilled in theart will recognize many modifications may be made to this configurationwithout departing from the scope or spirit of the claimed subjectmatter.

Although the subject matter has been described in language specific tostructural features and/or methodological acts, it is to be understoodthat the subject matter defined in the appended claims is notnecessarily limited to the specific features or acts described above.Rather, the specific features and acts described above are disclosed asexample forms of implementing the claims.

1. A method for determining an orientation of a portable electronicdevice, comprising: sensing a rotational rate of the device using afirst rotational sensor; sensing a total acceleration including agravitational component and linear acceleration component of the deviceusing a second inertial sensor; filtering data obtained by the secondinertial sensor using a variable low pass filter which filters the databy an amount that depends at least in part on the sensed rotational rateof the device; deriving an orientation of the device at successiveperiods of time based in part on the rotational rate sensed at eachtime; and applying a correction factor to the derived orientation of thedevice based at least in part on the sensed gravitational component andthe linear acceleration component of the filtered data obtained by thesecond inertial sensor when the second inertial sensor senses a totalacceleration that is equal to the gravitational component within apredefined threshold to thereby obtain a corrected orientation of thedevice at a subset of the successive periods of time.
 2. The method ofclaim 1 further comprising reducing the amount of filtering performed onthe data obtained by the second inertial sensor as the sensed rotationalrate of the device increases.
 3. The method of claim 1 in which thecorrected orientation of the device is obtained in body coordinatesassociated with the device and further comprising transforming thecorrected orientation of the device in body coordinates to Earthcoordinates.
 4. The method of claim 1 in which applying the correctionfactor includes adjusting the derived orientation in response to afeedback control signal from the second inertial sensor to therebyobtain the corrected orientation of the device.
 5. The method of claim 4in which the feedback control signal is generated in a feedback controlprocess that includes a proportional function and an integral function.6. The method of claim 5 further comprising entering a fast correctionmode of the feedback control process when valid data from the firstinertial sensor is unavailable, the feedback control process in saidfast correction mode having a faster proportional response relative to anominal proportional response used when valid data from the firstinertial sensor is available.
 7. The method of claim 5 in which the fastcorrection mode is entered and exited at a rate greater than a linearrate.
 8. The method of claim 7 in which the rate greater than the linearrate is a hyperbolic rate.
 9. The method of claim 1 further comprisingdetermining a geographic direction using a magnetometer, whereinapplying the correction factor is further based on the geographicdirection that is determined.
 10. The method of claim 9 furthercomprising initializing the determination of the orientation of thedevice using absolute orientation information obtained from the secondinertial sensor and the magnetometer.
 11. A portable electronic device,comprising: at least first and second inertial sensors for sensingmotion of the device; one or more processors for executingcomputer-executable instructions; and one or more computer-readablestorage media for storing the machine-executable instructions, theinstructions when executed by the one more processors implementingprocessing logic that, based on motion data derived from the firstinertial sensor in response to physical movement of the electronicdevice, is capable of tracking an orientation of the device, saidtracked orientation having an error associated therewith such that theprocessing logic periodically reduces the error using a feedback controlsignal of a PID control process based on additional motion data obtainedfrom the second inertial sensor, wherein, if said processing logicdetermines that the rotational motion data obtained from the firstinertial sensor is invalid or unavailable while tracking the orientationof the device, the processing logic is capable of suppressing anintegral function and causing a proportional function of the PID controlprocess to increase its response rate until the rotational motion datais again available and valid.
 12. The portable electronic device ofclaim 11 in which the first inertial sensor is a gyroscope providingrotational motion data and the orientation derived from the rotationalmotion is a relative orientation of the device.
 13. The portableelectronic device of claim 12 in which the processing logic initializesthe rotation motion using the feedback control signal to obtain anabsolute rotation motion of the device.
 14. The portable electronicdevice of claim 12 in which the second inertial sensor is anaccelerometer sensing a total acceleration that includes a gravitationalcomponent and a linear acceleration component.
 15. The method of claim14 further comprising filtering data obtained by the second inertialsensor using a variable low pass filter which filters the data by anamount that depends at least in part on the sensed rotational rate ofthe device.
 16. The method of claim 15 further comprising reducing theamount of filtering performed on the data obtained by the secondinertial sensor as the sensed rotational rate of the device increases.17. The method of claim 11 in which the processing logic suppresses theintegral function and causes the proportional function to increase at arate greater than a linear rate.
 18. One or more computer-readablestorage media containing instructions which, when executed by one ormore processors disposed in a mobile electronic device, perform a methodcomprising the steps of: determining an orientation of a mobileelectronic device using at least a first inertial motion sensor withwhich the mobile electronic device is equipped; providing a correctionfactor to the orientation of the mobile electronic device using afeedback control signal based on motion data obtained from at least asecond inertial motion sensor to reduce drift in motion data obtainedfrom the first inertial sensor; responsive to a loss of valid motiondata from the first inertial motion sensor, increasing a rate at whichthe correction factor is provided to the orientation of the mobileelectronic device.
 19. The computer-readable storage media of claim 18in which the first inertial motion sensor is a gyroscope detectingrotational motion of the device and the second inertial motion sensor isan accelerometer sensing a total acceleration that includes agravitational component and a linear acceleration component saidcorrection factor being provided to the orientation of the mobileelectronic device when the accelerometer senses a total accelerationequal to the gravitational component within a predefined threshold. 20.The computer-readable storage media of claim 18 in which the secondinertial motion sensor is an accelerometer further comprising filteringmotion data obtained by the accelerometer using a variable low passfilter which filters the motion data obtained by the accelerometer by anamount that depends at least in part on the sensed rotational rate ofthe device.